Root cause: illegal connections are not cleaned up when grpc call is failed
Fix: We remove useless connections when grpc call fails with unavailable or cancelled codes
Related issue: pingcap/tiflash#9806

Summary by CodeRabbit

• Bug Fixes

  • Improved handling and propagation of gRPC errors, ensuring failed RPCs mark unhealthy endpoints for removal.
  • More reliable cleanup of unavailable connections during background scans.

• New Features

  • Background health monitoring and automatic removal of bad connections.
  • RPC client now runs as a background task and is stopped earlier during cluster shutdown for more predictable lifecycle behavior.